--Autonomous agents capable of diverse object manipulations should be able to acquire a wide range of manipulation skills with high reusability. Although advances in deep learning have made it increasingly feasible to replicate the dexterity of human teleoperation in robots, generalizing these acquired skills to previously unseen scenarios remains a significant challenge. In this study, we propose a novel algorithm, Gaze-based Bottleneck-aware Robot Manipulation (GazeBot), which enables high reusability of the learned motions even when the object positions and end-effector poses differ from those in the provided demonstrations. By leveraging gaze information and motion bottlenecks--both crucial features for object manipulation--GazeBot achieves high generalization performance compared with state-of-the-art imitation learning methods, without sacrificing its dexterity and reactivity. Furthermore, the training process of GazeBot is entirely data-driven once a demonstration dataset with gaze data is provided. Videos and code are available at https://crumbyrobotics.github.io/gazebot. Recent advancements utilizing powerful neural networks such as Transformers have made deep imitation learning increasingly capable of reproducing dexterity to a certain extent [29, 5, 14]. However, significant issues persist regarding their generalization capabilities. Although generalization in object manipulation occurs at multiple levels, even the most fundamental aspects, such as changes in object position and the end-effector pose, are known to cause drastic reductions in success rates with variations of just a few centimeters [4]. For instance, ACT [29], a model recognized for its strong dexterous capabilities, has only been validated with objects placed on white tape with an accuracy of approximately 5 cm. Although ACT demonstrated high success rates under these specific conditions in our experiments, it was unable to reach objects placed in unseen positions (Figure 1), highlighting the poor generalization capabilities acquired through this method.